Systems and methods for three-dimensional visualization during robotic surgery

ABSTRACT

An autostereoscopic three-dimensional display system for surgical robotics has an autostereoscopic three-dimensional display configured to receive and display video from a surgical robotics camera, and a first sensor assembly and a second sensor assembly. A processor is configured to detect and track an eye position or a head position of a user relative to the display based on processing output data of the first sensor assembly, and to detect and track a gaze of the user based on processing output data of the second sensor assembly. The processor further is configured to modify or control an operation of the display system based on the detected gaze of the user. A spatial relationship of the display also can be automatically adjusted in relation to the user based on the detected eye or head position of the user to optimize the user&#39;s visualization of three-dimensional images on the display.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims the benefit of U.S. ProvisionalApplication Ser. No. 62/693,850 filed on Jul. 3, 2018.

INCORPORATION BY REFERENCE

U.S. Provisional Application No. 62/693,850, which was filed on Jul. 3,2018, is specifically incorporated by reference herein as if set forthin its entirety.

TECHNICAL FIELD

This disclosure relates generally to the field of surgical robotics and,more particularly, to display systems for use with surgical roboticsystems for visualizing the surgical site.

BACKGROUND

Minimally-invasive surgery (MIS), such as laparoscopic surgery, involvestechniques intended to reduce tissue damage during a surgical procedure.For example, laparoscopic procedures typically involve creating a numberof small incisions in the patient (e.g., in the abdomen), andintroducing one or more tools and at least one endoscopic camera throughthe incisions into the patient. The surgical procedures are thenperformed by using the introduced tools, with the visualization aidprovided by the camera.

Generally, MIS provides multiple benefits, such as reduced patientscarring, less patient pain, shorter patient recovery periods, and lowermedical treatment costs associated with patient recovery. In someembodiments, MIS may be performed with surgical robotic systems thatinclude one or more robotic arms for manipulating surgical instrumentsbased on commands from an operator. For example, an operator may providecommands for manipulating surgical instruments, while viewing an imagethat is provided by a camera and displayed on a display to the user.However, conventional display systems fall short in enabling effectivecontrol of the display systems or of surgical robotic systems.Furthermore, conventional display systems generally providetwo-dimensional (2-D) surgical image data to the user, and currentthree-dimensional (3-D) displays typically require the user to wearglasses or additional, similar wearable components (e.g., withpolarizing filters or dynamic shutters) for visualization ofthree-dimensional images. Such glasses and additional wearablecomponents, however, may be problematic to use and handle in surgical orsterile environments. Thus, there is a need for improved 3-D displaysystems that enable a user to better visualize the surgical site duringrobotic surgery.

SUMMARY

Generally, a three-dimensional display system for use with a surgicalrobotic system can include a three-dimensional display configured toreceive and display video from a surgical robotics camera, such as anendoscopic camera. The display system can include a plurality of sensorassemblies having a first sensor assembly and a second sensor assembly.The first sensor assembly and the second sensor assembly can be coupledto or integrally formed with the display. The display system can includea processor or controller configured to detect and track an eye positionor a head position of a user relative to the display based on processingoutput data of the first sensor assembly. The processor or controlleralso can be configured to detect and track a gaze of the user based onprocessing output data of the second sensor assembly.

The processor or controller further is configured to modify or controlan operation of the display system based on the detected and trackedgaze of the user, for example, to facilitate control of the displaysystem with the user's eyes or eye motions. In addition, a spatialrelationship of the display can be automatically adjusted in relation tothe user based on the detected eye or head position of the user. Forexample, a distance or orientation between the detected eye or headposition and the display can be automatically (e.g., without requiringdeliberate user input) updated to adjust the user's visualization ofthree-dimensional image data from the surgical robotics camera on thedisplay.

In some variations, the display can include a panel display or monitor,such as an LCD, LED, plasma, or other suitable flat, curved, orotherwise shaped panel display or monitor, having a plurality of pixelsfor displaying two or three-dimensional images. The display further caninclude one or more layers at least partially positioned over the paneldisplay and configured to facilitate a user's visualization ofthree-dimensional images on the panel display. The one or more layerscan include a polarizing filter, a pattern retarder, or dynamic shuttersthat allow users to uses glasses or other wearable components to view orvisualize the three-dimensional images on the panel display.Alternatively, the one or more layers can include layers of micro-lensesthat can at least partially cover the plurality of pixels of the paneldisplay. The layer(s) of micro-lenses further can be positioned ordisposed in relation to the plurality of pixels to facilitate orotherwise allow the user's visualization or perception ofthree-dimensional images on the panel display, without the use ofthree-dimensional glasses or other additional wearable or similarcomponents worn by a user. The display further can include a protectivelayer at least partially disposed over or sealing off the layer(s) ofthe panel display. The protective layer can be bonded to layer(s) or thepanel display using an adhesive, such as an optically clear adhesive orother suitable adhesive. An additional protective layer can be providedon the display panel, e.g., between the one or more layers includingmicro-lenses and the display panel.

In some variations, the first sensor assembly can include at least onecamera, such as a stereo camera, an infrared camera, or other suitablecamera that does not filter infrared light, e.g., to allow for detectionand tracking of a head or eye position of a user (e.g., an xyz positionof the user's head or eye position in relation to an origin or originalposition or to the display). The second sensor assembly can include oneor more cameras and a plurality of strobes or strobe lights, e.g., toallow for illumination of and detection and tracking of an iris oririses of the user's eyes.

In addition, a seat assembly can be provided with the display system.The seat assembly can have a seat in which a user is to sit or otherwiseengage, while the user is viewing the display. The seat assembly alsocan include a movable or adjustable seat support assembly that isconnected to and at least partially supports the seat. The processor orcontroller can automatically generate and send one or more signals orother output data to an actuator subsystem of the seat support assemblyto adjust or update a position or orientation of the seat based uponreceived output data from the first or second sensor assemblies. Forexample, the position or orientation of the seat can be adjusted basedon the detected and tracked eye or head position of the user to optimizethe user's visualization of three-dimensional images on the display.

The display system also can include a movable or adjustable displaysupport assembly connected to and supporting the display. The processoror controller can automatically generate and send one or more signals orother output data to an actuator subsystem of the movable or adjustabledisplay support assembly to adjust or update a position or anorientation of the display based upon received output data from thefirst or second sensor assemblies. For example, the position ororientation of the display can be adjusted based on the detected andtracked eye or head position of the user to optimize the user'svisualization of three-dimensional images from the surgical roboticscamera on the display.

In one example, the position or orientation of the seat or the displaycan be automatically adjusted or changed such that the user's head oreyes are located at a predetermined distance from or orientation inrelation to the display.

In some variations, the processor or controller can be in communicationwith the surgical robotic system. The processor further can be operableto send a signal or other output data to the surgical robotic system,e.g., to a controller thereof, for control of the surgical roboticsystem based on received output data from the first or second sensorassemblies. For example, when the gaze of the user is not directedtowards the display, e.g., for a predetermined time interval, control ofone or more operations of the surgical robotic system (e.g., operationsof robotic arms or surgical instruments) may be paused or otherwisedisabled.

Additionally, an endoscopic image or other suitable image of a surgicalsite from the surgical robotics camera may be displayed on the display,e.g., as part of a GUI or display window on the display. Control panelsor side panels having a plurality of icons or images additionally oralternatively can be displayed on the display. For example, control orside panels can be positioned to the left and right of the primarydisplay or window on the display. The plurality of icons or images canbe related to applications for the display system or the surgicalrobotic system. The detected and tracked gaze of the user further can beused to initiate or control the applications in the control/side panels.For example, a user can focus their gaze on the images or icons shownthe control or side panels to trigger application interactions (e.g., tostart and stop a timer application, initiate or control an x-ray viewingtool, enlarge a view, or to initiate or control other suitableapplications).

In some variations, a position or orientation of the surgical roboticscamera can be dynamically or continuously updated based on the detectedand tracked gaze of the user. For example, the position of the surgicalrobotics camera can be automatically updated such that an area or pointsubstantially focused on by the user's gaze, e.g., an area or pointwithin the primary display or window showing the endoscopic image, issubstantially centered on the display. In one embodiment, when theprocessor or controller determines that the detected gaze of the user isdirected at an area or point that is spaced apart from the center of thedisplay, the processor or controller generates and sends a signal to thesurgical robotics camera to adjust the position or orientation of thesurgical robotics camera such that the area or point at which the user'sgaze is directed or focused on is moved to the center of the display.

Furthermore, a method for three-dimensional visualization during roboticsurgery can be provided. The method can be performed by a digitalprogrammed processor executing instructions stored in a computerreadable memory. The method can include receiving and displaying videofrom a surgical robotics camera on a three-dimensional display. Themethod further can include detecting and tracking a head position or aneye position of a user relative to the display based on processingoutput data of a first sensor assembly, and detecting and tracking agaze of the user based on processing output data of a second sensorassembly. The detected and tracked gaze of the user can be used tofacilitate control or modify operations of a display system or asurgical robotic system. In addition, the method can includeautomatically (e.g., without requiring deliberate user input) signalingan actuator subsystem to adjust or update a spatial relationship of thedisplay in relation to the user based on the detected eye or headposition of the user to optimize the user's visualization ofthree-dimensional images from the surgical robotics camera on thedisplay.

In some variations, a position or orientation of the display or a seatassembly, which is configured to be sat in or otherwise engaged by theuser when viewing the display, can be automatically adjusted or modifiedbased upon the detected and tracked head or eye position of the user.

In further variations, an operation(s) of the surgical robotic system ordisplay system also can be modified or otherwise controlled based on thedetected and tracked gaze of the user. For example, the processor orcontroller can automatically signal an actuator subsystem of thesurgical robotics camera to update or alter a position of a lens of thesurgical robotics camera based on the gaze of the user. Morespecifically, the position or orientation of the surgical camera can beautomatically altered or updated such that the point or area focused onby the user's gaze is substantially centered on/along the display.Further, when the detected and tracked gaze of the user is directed atan image or icon that is related to an application, e.g., an image oricon of a control or side panel displayed on the display, theapplication can be initiated or otherwise controlled. Still further,when the detected and tracked gaze of the user is not directed at thedisplay, e.g., for a predetermined time interval, an operation of thesurgical robotic system can be disabled.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial view of an example surgical robotic system in anoperating arena.

FIG. 2 shows a schematic view of an exemplary console for use with thesurgical robotics system.

FIG. 3 shows an exploded side view of an exemplary display or monitor.

FIG. 4 shows an exemplary display system for use with the surgicalrobotics system.

DETAILED DESCRIPTION

Non-limiting examples of various aspects and variations of the inventionare described herein and illustrated in the accompanying drawings.

Referring to FIG. 1, this is a pictorial view of an example surgicalrobotic system 1 in an operating arena. The robotic system 1 includes auser console 2, a control tower 3, and one or more surgical robotic arms4 at a surgical robotic platform 5, e.g., a table, a bed, etc. Thesystem 1 can incorporate any number of devices, tools, or accessoriesused to perform surgery on a patient 6. For example, the system 1 mayinclude one or more surgical tools 7 used to perform surgery. A surgicaltool 7 may be an end effector that is attached to a distal end of asurgical arm 4, for executing a surgical procedure.

Each surgical tool 7 may be manipulated manually, robotically, or both,during the surgery. For example, the surgical tool 7 may be a tool usedto enter, view, or manipulate an internal anatomy of the patient 6. Inone embodiment, the surgical tool 7 is a grasper that can grasp tissueof the patient. The surgical tool 7 may be controlled manually, by abedside operator 8; or it may be controlled robotically, via actuatedmovement of the surgical robotic arm 4 to which it is attached. Therobotic arms 4 are shown as a table-mounted system, but in otherconfigurations the arms 4 may be mounted in a cart, ceiling or sidewall,or in another suitable structural support.

Generally, a remote operator 9, such as a surgeon or other operator, mayuse the user console 2 to remotely manipulate the arms 4 or the attachedsurgical tools 7, e.g., teleoperation. The user console 2 may be locatedin the same operating room as the rest of the system 1, as shown inFIG. 1. In other environments, however, the user console 2 may belocated in an adjacent or nearby room, or it may be at a remotelocation, e.g., in a different building, city, or country. The userconsole 2 may comprise a seat 10, foot-operated controls 13, one or morehandheld user input devices, UID 14, and at least one user display 15that is configured to display, for example, a view of the surgical siteinside the patient 6. In the example user console 2, the remote operator9 is sitting in the seat 10 and viewing the user display 15 whilemanipulating a foot-operated control 13 and a handheld UID 14 in orderto remotely control the arms 4 and the surgical tools 7 (that aremounted on the distal ends of the arms 4.)

In some variations, the bedside operator 8 may also operate the system 1in an “over the bed” mode, in which the beside operator 8 (user) is nowat a side of the patient 6 and is simultaneously manipulating arobotically-driven tool (end effector as attached to the arm 4), e.g.,with a handheld UID 14 held in one hand, and a manual laparoscopic tool.For example, the bedside operator's left hand may be manipulating thehandheld UID to control a robotic component, while the bedsideoperator's right hand may be manipulating a manual laparoscopic tool.Thus, in these variations, the bedside operator 8 may perform bothrobotic-assisted minimally invasive surgery and manual laparoscopicsurgery on the patient 6.

During an example procedure (surgery), the patient 6 is prepped anddraped in a sterile fashion to achieve anesthesia. Initial access to thesurgical site may be performed manually while the arms of the roboticsystem 1 are in a stowed configuration or withdrawn configuration (tofacilitate access to the surgical site.) Once access is completed,initial positioning or preparation of the robotic system 1 including itsarms 4 may be performed. Next, the surgery proceeds with the remoteoperator 9 at the user console 2 utilizing the foot-operated controls 13and the UIDs 14 to manipulate the various end effectors and perhaps animaging system, to perform the surgery. Manual assistance may also beprovided at the procedure bed or table, by sterile-gowned bedsidepersonnel, e.g., the bedside operator 8 who may perform tasks such asretracting tissues, performing manual repositioning, and tool exchangeupon one or more of the robotic arms 4. Non-sterile personnel may alsobe present to assist the remote operator 9 at the user console 2. Whenthe procedure or surgery is completed, the system 1 and the user console2 may be configured or set in a state to facilitate post-operativeprocedures such as cleaning or sterilization and healthcare record entryor printout via the user console 2.

In one embodiment, the remote operator 9 holds and moves the UID 14 toprovide an input command to move a robot arm actuator 17 in the roboticsystem 1. The UID 14 may be communicatively coupled to the rest of therobotic system 1, e.g., via a console computer system 16. The UID 14 cangenerate spatial state signals corresponding to movement of the UID 14,e.g. position and orientation of the handheld housing of the UID, andthe spatial state signals may be input signals to control a motion ofthe robot arm actuator 17. The robotic system 1 may use control signalsderived from the spatial state signals, to control proportional motionof the actuator 17. In one embodiment, a console processor of theconsole computer system 16 receives the spatial state signals andgenerates the corresponding control signals. Based on these controlsignals, which control how the actuator 17 is energized to move asegment or link of the arm 4, the movement of a corresponding surgicaltool that is attached to the arm may mimic the movement of the UID 14.Similarly, interaction between the remote operator 9 and the UID 14 cangenerate for example a grip control signal that causes a jaw of agrasper of the surgical tool 7 to close and grip the tissue of patient6.

The surgical robotic system 1 may include several UIDs 14, whererespective control signals are generated for each UID that control theactuators and the surgical tool (end effector) of a respective arm 4.For example, the remote operator 9 may move a first UID 14 to controlthe motion of an actuator 17 that is in a left robotic arm, where theactuator responds by moving linkages, gears, etc., in that arm 4.Similarly, movement of a second UID 14 by the remote operator 9 controlsthe motion of another actuator 17, which in turn moves other linkages,gears, etc., of the robotic system 1. The robotic system 1 may include aright arm 4 that is secured to the bed or table to the right side of thepatient, and a left arm 4 that is at the left side of the patient. Anactuator 17 may include one or more motors that are controlled so thatthey drive the rotation of a joint of the arm 4, to for example change,relative to the patient, an orientation of an endoscope or a grasper ofthe surgical tool 7 that is attached to that arm. Motion of severalactuators 17 in the same arm 4 can be controlled by the spatial statesignals generated from a particular UID 14. The UIDs 14 can also controlmotion of respective surgical tool graspers. For example, each UID 14can generate a respective grip signal to control motion of an actuator,e.g., a linear actuator, that opens or closes jaws of the grasper at adistal end of surgical tool 7 to grip tissue within patient 6.

In some aspects, the communication between the platform 5 and the userconsole 2 may be through a control tower 3, which may translate usercommands that are received from the user console 2 (and moreparticularly from the console computer system 16) into robotic controlcommands that transmitted to the arms 4 on the robotic platform 5. Thecontrol tower 3 may also transmit status and feedback from the platform5 back to the user console 2. The communication connections between therobotic platform 5, the user console 2, and the control tower 3 may bevia wired or wireless links, using any suitable ones of a variety ofdata communication protocols. Any wired connections may be optionallybuilt into the floor or walls or ceiling of the operating room. Therobotic system 1 may provide video output to one or more displays,including displays within the operating room as well as remote displaysthat are accessible via the Internet or other networks (e.g., therobotic system 1 can include one or more endoscopic cameras that providevideo output or other suitable image data to the displays). The videooutput or feed may also be encrypted to ensure privacy and all orportions of the video output may be saved to a server or electronichealthcare record system.

FIG. 2 shows a schematic view of an exemplary user console 2. As shownin FIG. 2, a display system 140 can be provided for use with the userconsole 2 and the surgical robotic system 1. The display system 140includes an autostereoscopic, three-dimensional monitor or display 142configured display three-dimensional (3D) or two-dimensional (2D)information to a user. The monitor 142 may display various informationassociated with the surgical procedure (e.g., an endoscopic camera viewof the surgical site, static images, GUIs, etc.) or surgical roboticsystem (e.g., status, system settings), or other suitable information inthe form of 2D and 3D video, image data, text, graphical interfaces,warnings, controls, indicator lights, etc. The monitor 142 as describedherein further may enable the user to interact with displayed contentusing eye movements or other suitable gestures of the user for controlof the display system and operation of other instruments such as thosein the surgical robotic system.

FIG. 2 additionally shows that the display system 140 includes aplurality of sensor assemblies 144 and 146 including a first or head (oreye) tracking sensor assembly 144, and a second or gaze tracking sensorassembly 146. The first and second sensor assemblies 144 and 146 can beattached to, or in some variations integrally formed with, the monitor142. For example, the first sensor assembly 144 can be connected to anupper or top portion of the monitor 142 and the second sensor assembly146 can be connected to a lower or bottom portion of the monitor 142, asgenerally shown in FIG. 2. However, in the alternative, the secondsensor assembly 146 can be attached to the top portion of the monitor142 and the first sensor assembly 144 can be attached to the bottomportion of the monitor 142, or both the first and second sensorassemblies 144 and 146 can be attached to the top or bottom portions ofthe monitor 142, or the first or second sensor assemblies 144 and 146can be attached to side portions of the monitor 142. The first or secondsensor assemblies 144 and 146 also can be coupled to or incorporatedwith other suitable components or parts of or near the console 2,without departing from the scope of the present disclosure.

As further shown in FIG. 2, the monitor 142 may be supported by a poweradjustable monitor support assembly 150. The monitor 142 may bepositioned proximate or near the seat 10 to enable a user to view themonitor while the user is seated in or otherwise engaged by the seat 10.For example, the support assembly 150 may have a support or column 152positioned in front or forward of the seat 10, which support or column152 at least partially supports the monitor 142. In one variation, themonitor 142 is connected to the support 152 by an adjustable mountassembly 154 including an actuator subsystem with one or more actuators158, 162, 166 that enable automatic adjustment of a position ororientation of the monitor 152 (e.g., based upon output data receivedfrom the first or second sensor assemblies 144 or 146). The monitor 142further can include one or more sensors (e.g., position sensors, motionsensors, accelerometers, etc.) attached to the monitor 142 thatfacilitate the detection and tracking of positions or orientations ofthe monitor.

The mount assembly 154 can enable translation or rotational movement ofthe monitor 142 for up to six degrees of freedom including, e.g., tilt,yaw, rotation, front-to-back movement, side-to-side movement, andup-and-down movement. For example, the mount assembly 154 can include aslidable support portion or member 156 coupled to the monitor 142. Theslidable support portion 156 further can be driven by one or moreactuators 158 (e.g., motors, hydraulic actuators, pneumatic actuators,etc.) for up-down and side-to-side translation of the monitor 142. Themounting assembly 154 further can include one or more telescopingportions or sections 160 or other suitable portions or components thatare driven by one or more actuators 162 to enable forward and backwardmovement of the monitor 142 (i.e., movement of the monitor 142 towardsand away from the seat 10, e.g., to vary a distance between the seat 10and the monitor 142). The telescoping portions 160 can connect themonitor 142 to the slidable support portion 156. The mounting assembly154 also can include a pivotable connection 164 (e.g., a swivel fixture,ball joint, pivoting feature, etc.) connecting the monitor 142 to thetelescoping portions 160. Movement of the monitor 142 about thepivotable connection 164 can be driven by an actuator 166 (e.g., motors,hydraulic actuators, pneumatic actuators, etc.) to enable tilt, yaw, androtation of the monitor 142. The mounting assembly 154 further can allowfor manual adjustment of the position or orientation of the monitor 142.

FIG. 2 further shows that the seat 10 can be supported by a poweradjustable seat support assembly 165. The power adjustable seat supportassembly 165 can have an actuator subsystem including actuators 169/171that drive movement of the seat 10. As shown in FIG. 2, the seat supportassembly 165 includes seat support 167 having a single pillar at leastpartially supporting the seat 10, but in other examples, the seatsupport 167 may comprise two or more pillars. The seat support 167 canbe angled posteriorly in relation to the monitor 142, but in othervariations, may be angled vertically straight upward or tiltedanteriorly. In some variations, the seat 10 can be moveably oradjustably mounted to the seat support 167. For example, the seat 10 canrotate, tilt, recline, etc. in relation to the support 167 to enableadjustment of a position or orientation of the seat 10 in relation tothe monitor 142 (e.g., such that a position of the user's head or eyescan be automatically adjusted in relation to the monitor 142 to optimizevisualization or perception of three-dimensional images thereon). Theseat assembly 165 further can have one or more actuators 169 (e.g.,motors, hydraulic actuators, pneumatic actuators, etc.) forautomatically driving rotation, tilting, reclining, etc. of the seat 10(e.g., in response to output data from the first or second sensorassemblies 144/146).

In some variations, the seat 10 further is moveable along the support167 (e.g., to move the seat 10 up and down and forward and backward inrelation to the monitor 142). For example, an actuator 171 (e.g., amotor, a hydraulic actuator, a pneumatic actuator, etc.) can drivemovement of the seat 10 along the support 167 (e.g., in response tooutput data from the first or second sensor assemblies 144/146). Inaddition, or in the alternative, the seat support 167 may be configuredto change its angle or orientation, or to translate in the forward orrearward directions or in the lateral directions. In some furthervariations, the seat support 167 may be configured to telescope orotherwise extend or retract longitudinally or generally vertically. Theseat support assembly 165 further may allow for manual adjustment theposition or orientation of the seat 10.

FIG. 3 shows a cross-sectional view of the display or monitor 142. Thedisplay or monitor 142 can include a flat, curved, or otherwise shapedpanel display 170, such as an LCD, LED, plasma, or other suitable paneldisplay, having a plurality of pixels for displaying two orthree-dimensional images. The display 142 further can include one ormore layers 172/174 at least partially overlaying, or otherwisedisposed/positioned over, the display 142. The layers 172/174 areconfigured to facilitate a user's visualization of three-dimensionalimages on the display 142. In one embodiment, the layers 172/174 caninclude micro-lenses that can be at least partially positioned over theplurality of pixels of the panel display 170 to facilitate or otherwiseallow for the user's visualization or perception of three-dimensionalimages on the panel display 170. In one embodiment, the pixels of thepanel display 170 can display a left eye image and a right eye imagethat are continuously and/or dynamically interleaved, and the layers 172or 174 can enable the user to visualize or perceive the left eye imageand the right eye image as a single three-dimensional image, without theuse of three-dimensional glasses or other additional wearable or similarcomponents on the user. In the alternative, the one or more layers172/174 can include polarizing filters, a patterned retarder, or dynamicshutters, and a user may use three-dimensional glasses or other similarwearable components to view or visualize three-dimensional images on thedisplay.

The display 142 further can include a protective layer 176 at leastpartially covering or sealing off the layer(s) 172/174 or the paneldisplay 170. The protective layer 176 may seal off and protect thelayers 172/174 and panel display 170 such that the monitor 142 issuitable for use in a surgical environment. For example, the protectivelayer 176 may allow for sterilization or cleaning of the display (e.g.,with cleaning chemicals, such as alcohol-based or chlorine-basedcleaners) without damage to the micro-lenses 172/174. In one embodiment,the protective layer 176 can include surgical-grade glass or othersurgical-grade materials (e.g., surgical-grade plastics or othersuitable composite materials). The protective layer 176 further can havea thickness in the range of approximately 1 mm to approximately 3.0 mm,such as approximately 2.0 mm or other suitable integer and non-integernumbers therebetween. Thicknesses of less than 1.5 mm or greater than3.0 mm can be employed, however, without departing from the scope of thepresent disclosure. Additionally, or in the alternative, at least oneadditional protective layer 177 can be provided on the panel display 170(e.g., between the panel display 170 and the layer(s) 170). Theadditional protective layer 177 can have a thickness of up to 1.0 mm,such as approximately 0.3 mm, and can be formed from plastic, glass, orother suitable material.

The protective layer 176 can be bonded to one or both of the layers172/174 or the panel display 170 using an adhesive 178 (e.g., anoptically clear adhesive or other suitable adhesive or glue). One ormore spacers 179 further may be provided between the protective layer176 and the layers 172/174 or the panel display 170. The spacers 179 canbe positioned along a boundary of the protective layer 176 at equallyspaced intervals, though in some variations the spacers 179 can bedisposed intermittently or sporadically about the protective layer 176.The spacers 179 can prevent damage to the layers 174/176 duringformation of the monitor, e.g., during application and bonding of theprotective layer 176.

As shown in FIG. 4, in some variations, the first sensor assembly 144can include one or more sensors 200. The sensors 200 can include astereo camera(s), an infrared camera(s), or other suitable camera(s) 202that does not filter infrared light. In one embodiment, the camera(s)202 can include one Intel® Real Sense Camera as provided by Intel Corp.of Santa Clara, Calif. The sensors 200 additionally or alternatively caninclude other types of cameras, e.g., color cameras, or other suitablesensing devices, without departing from the scope of the presentdisclosure. Signals or output information from the first sensor assembly144 can be received and processed, e.g., by a controller or processor incommunication with the first sensor assembly 144, to facilitate orotherwise allow for detection and tracking of a head or eye position ofa user. For example, the first sensor assembly 144 can be used fordetecting and tracking an xyz position of a user's head, eye, or eyes,e.g., in relation to an origin or an original position, such that aposition, e.g., a distance, of the user's head or eyes can becontinuously determined in relation to the monitor 142.

In addition, in some variations, the second sensor assembly 146 includesone or more sensors 210, such as one or more cameras 212, and one ormore strobes or strobe lights 214, e.g., that flash light to facilitatedetection and tracking of a gaze of a user by the camera(s) 212. Thegaze of the user is detected based on a position or a movement of atleast one iris of the user's eyes and includes an area or point at whichthe user is looking or substantially focused (e.g., an area or point onthe monitor or an area or point off/away from the monitor). In oneembodiment, the strobe(s) 214 can be configured to provide multipleflashes of light per second, e.g., flashes of light at a frequency inthe range of approximately 80 Hz to approximately 100 Hz, such asapproximately 90 Hz or other suitable frequency. The camera 212 includesa high-speed camera that is configured to capture the illuminated (e.g.,by the strobes 214) and unilluminated irises of the user (e.g., suchthat the processor receiving and processing output data from the camera212 can detect and track a user's irises to determine a point or area atwhich the user is looking or substantially focused on). The lightflashes from the strobes 214 further may assist in perception of theuser's eye or head position with the first sensor assembly 144, e.g.,during low light conditions.

It should be understood that although some specific examples of sensortypes, sensor locations, and sensor functions in the display system havebeen discussed above, a wide variety of other sensors and sensor typesmay additionally or alternatively be located throughout the variouscomponents of the display system in order to capture information aboutthe user or for receiving user input as interactive user controls.

An example of a graphical user interface (GUI) to be displayed on themonitor 142 is shown in FIG. 4. For example, the GUI may display adisplay portion or display window 180 showing endoscopic image or othersuitable surgical image data (e.g., from an endoscopic camera or othersuitable surgical robotics camera placed inside the patient). The GUImay further include control panels or side panels 182 including one ormore images or icons 184 related to one or more applications related tothe surgical robotic system 1 (e.g., a timer application, an x-rayimaging tool, etc.). The control panel(s) 182 also can include othersuitable information, such as one or more medical images (e.g.,pre-operative images of patient tissue), patient data (e.g., name,medical record number, date of birth, various suitable notes, etc.),tool information (e.g., a left or right tool number, a left or righttool name, a left or right tool function, etc.). Other suitable GUIs orother display content may appear on the monitor without departing fromthe present disclosure. Various user interactions (e.g., the user's gazeor eye or head movements) also may cause changes to the displayedcontent type, as well as interaction with the applications as furtherdescribed below.

The display system 140 generally includes or is in communication with aprocessor or controller configured to detect and track a head positionor an eye position of a user relative to the monitor 142 based onprocessing output data of the first sensor assembly 144. In somevariations, a spatial relationship between the monitor 142 and a user(e.g., a user sitting in seat 10) can be adjusted based on the detectedeye or head position of the user, e.g., to optimize the user'svisualization or perception of three-dimensional image data from theendoscopic or surgical robotics camera on the monitor 142. A user'sperception of three-dimensional images on the monitor 142 may be optimalwhen the user's eyes are substantially centered with respect to themonitor 142 and spaced at a prescribed distance therefrom (e.g.,approximately 70 cm to approximately 90 cm, such as approximately 80 cmfrom the monitor). Thus, the position or orientation of the seat 10 orthe monitor 142 can be automatically (e.g., without requiring adeliberate user input) adjusted or changed to ensure that the user'shead or eyes are located and positioned at an optimal orientation orviewing distance in relation to the monitor.

In one embodiment, the processor or controller can be in communicationwith the seat actuators 169 and 171 or the monitor actuators 158, 162,166 and can automatically provide signals or information to seatactuators 169/171 or monitor 158, 162, 166 to adjust a position ororientation of the seat 10 or monitor 142 based upon processing outputsignals from the first or second sensor assemblies 144/166. For example,the processor or controller can determine a position of the user's head(or eyes) in relation to the monitor 142, and the processor canautomatically generate and send a signal(s) to the seat actuators 169 or171 or the monitor actuators 158, 162, or 166 to adjust or change theposition or orientation of the seat or monitor (e.g., the seat can bereclined, tilted, rotated, moved up or down, moved side to side, etc. orthe monitor can be tilted, yawed, rotated, moved front-to-back, movedside-to-side movement, moved up-and-down, etc.) based on the determinedposition of the user's head (or eyes), e.g., to optimize the user'svisualization of three-dimensional images from the surgical roboticscamera on the monitor. For example, the position or orientation of theseat or monitor can be adjusted such that the user's head (or eyes) issubstantially centered with respect to the monitor and is at aprescribed distance from the monitor for optimal viewing ofthree-dimensional images.

The processor or controller additionally, or alternatively, can generateand send signals to the monitor 142 to display instructions thereon formanual adjustment of the monitor 142 or the seat 10 to optimize theuser's perception or visualization of three dimensional image data onthe display.

Furthermore, the controller or processor is configured to detect thetrack the gaze of the user based on processing output data of the secondsensor assembly 146, and in some variations, operations of the displaysystem 140 or the surgical robotic system 1 can be modified orcontrolled based on the detected gaze of the user (e.g., to facilitatecontrol of the display system with the user's eyes or eye gestures or tostop or pause operations of the display system or surgical roboticsystem when the detected gaze of the user is directed away from themonitor).

In some variations, the processor or controller can be in communicationwith the surgical robotic system 1, and when the processor or controllerdetermines that the gaze of a user is not directed at the monitor 142(e.g., for a predetermined time period, such as approximately 3 secondsor up to approximately 5 seconds or more), the processor or controlleris operable to automatically send a signal(s) or other output data tothe surgical robotic system 1 or the display system 140 to activate ordisable one or more operations thereof (e.g., to disable or freezeoperation of one or more subsystems of the surgical system, such as therobotic arms 4 or the surgical tools 7, or to generate an alarm with thedisplay system).

In one embodiment, when the processor or controller determined that theuser's gaze is not directed at the monitor 142, e.g., for a prescribedtime period, such as when the user is distracted, falls asleep, etc.,the processor or controller automatically generates and sends one ormore signals to the surgical system 1 to freeze or pause operation ofthe robotic arms 4 or the surgical tools 7, e.g., to prevent injury to apatient being operated on. Further, when the processor or controllerdetermines that the user's gaze has returned to the monitor 142, theprocessor or controller may automatically generate and send one or moresignals to the surgical system to resume operation of the robotic arms 4or the surgical tools 7. However, the processor or controller mayrequire a specific user input (e.g., selection of an icon, a gesture,etc.) prior to sending the signal(s) for resuming operation of therobotic arms or surgical tools 7.

Additionally, or in the alternative, when the processor or controllerdetermines that the user's gaze is not directed at the monitor 142, theprocessor or controller may generate and send a signal(s) to the displaysystem 140 to activate one or more alarms or notifications to get theattention of the user or other suitable entity (e.g., a speaker of thedisplay system may play one or more audio sounds, the monitor maydisplay one or more images indicating that the user's gaze is notdirected at the monitor, or one or more vibrations or haptics of theseat or UIDs may be activated).

In some variations, the detected and tracked gaze of the user also canbe used to initiate or control the applications on the control/sidepanels 182. For example, a user can look at or focus on the one or moreimages 184 on the control or side panels 182 to trigger applicationinteractions. The user can initiate or close the applications, open theapplications in one or more new windows or pop-up windows, controlfeatures or operations of the applications, etc. by focusing on orlooking at one or more areas or points on the GUI or using othersuitable eye motions. In one example, the user can focus on or look atan image associated with a timer application shown on the control/sidepanels, e.g., to start and stop the timer. In another example, the usercan focus on or look at an image associated with an x-ray imaging toolto initiate the x-ray imaging tool (e.g., to open the x-ray imaging toolon one or more secondary or popup windows on the display). The user'sgaze further can be used to close the x-ray image tool (e.g., when theuser looks away or focuses on a close icon or image or other suitablefeature).

Additionally, a position or orientation of the surgical robotics cameraalso can be updated or adjusted based on the detected and tracked gazeof the user. In some variations, the position or orientation of thesurgical robotics camera can be continuously or dynamically updated(e.g., the controller or processor can automatically generate and sendsignals to an actuator subsystem of the surgical robotics camera totilt, rotate, or otherwise translate a lens of the surgical roboticscamera) such that an area or point on the monitor 142 that is beingfocused on by the user is substantially centered along the monitor 142(e.g., centered in relation to the horizontal axis and the vertical axisof the monitor) where perception or visualization of three-dimensionalimage data is optimal. That is, each time a user focuses on an area orpoint of the three-dimensional image data displayed in the displaywindow 180 that is not substantially centered along the display window180 (e.g., based on the user's detected gaze), the position ororientation of the surgical robotics camera can be updated or changedsuch that the area or point of the three-dimensional image data on whichthe user is focused is moved or otherwise adjusted along the displaywindow 180 so as to be substantially centered therealong.

For example, the user may initially focus on a point or area of thethree-dimensional image data that is substantially centered within thedisplay window 180, and when the user changes their focus or otherwiseredirects their gaze to a new area or point on the image data shown inthe display window 180 (e.g., the user looks at or focuses on an area orpoint that is proximate to or near an edge or corner of the displaywindow 180 or the user looks at or focuses on an area or point that isotherwise spaced apart from the original point or area in the center ofthe display window), the processor or controller may generate and sendone or more signals to the surgical robotics camera (or a controllerthereof) to automatically adjust the position or orientation of thesurgical robotics camera such that the new area or point of thethree-dimensional image data that is focused on by the user is moved oradjusted so as to be substantially centered within the display window180. In this way, the position or orientation of the surgical roboticscamera can be continuously or dynamically adjusted or otherwise updatedbased upon the determined gaze of the user such that the user's focus isdirected to be and remains generally centered along the display windowto facilitate optimal three-dimensional perception or visualization ofthe three-dimensional image data displayed therein.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the invention.However, it will be apparent to one skilled in the art that specificdetails are not required in order to practice the invention. Thus, theforegoing descriptions of specific embodiments of the invention arepresented for purposes of illustration and description. They are notintended to be exhaustive or to limit the invention to the precise formsdisclosed; obviously, many modifications and variations are possible inview of the above teachings. The embodiments were chosen and describedin order to best explain the principles of the invention and itspractical applications, and they thereby enable others skilled in theart to best utilize the invention and various embodiments with variousmodifications as are suited to the particular use contemplated.

What is claimed is:
 1. A three-dimensional display system for use in asurgical robotics system, comprising: an autostereoscopicthree-dimensional display configured to receive and display video from asurgical robotics camera, the display including one or more layers atleast partially positioned over the display and configured to facilitatea user's visualization of three-dimensional images on the display; aplurality of sensor assemblies that include a first sensor assemblyconfigured to capture information related to a head position of a user,and a second sensor assembly configured to capture information relatedto a gaze of the user; and a processor that detects and tracks the headposition of the user relative to the display based on processing outputdata of the first sensor assembly, and detects and tracks the gaze ofthe user based on processing output data of the second sensor assembly,the processor further modifies an operation of the display system basedon the detected gaze of the user, and automatically adjusts a spatialrelationship between the user and the display based on the detected headposition of the user to affect the user's visualization of thethree-dimensional images on the panel display.
 2. The display system ofclaim 1, wherein the one or more layers include at least one ofpolarizing filters, a pattern retarder, or dynamic shutters.
 3. Thedisplay system of claim 2 wherein the user is to use glasses or otherwearable components to view or visualize the three-dimensional images onthe display.
 4. The display system of claim 2, wherein the displayfurther comprises a protective layer covering the one or more layerspositioned on the display.
 5. The display system of claim 4, wherein theprotective layer includes surgical grade glass that is bonded to the oneor more layers or to the display by an optically clear adhesive.
 6. Thedisplay system of claim 1, wherein the one or more layers includes aplurality of micro-lenses positioned to facilitate the user'svisualization of three-dimensional images on the display, without theuse of glasses or other wearable components.
 7. The display system ofclaim 1, wherein the first sensor assembly and the second sensorassembly are physically attached to or integrated with the display. 8.The display system of claim 1, wherein the first sensor assemblycomprises an infrared camera, and wherein the second sensor assemblycomprises a high-speed camera and a plurality of strobes.
 9. The displaysystem of claim 1, further comprising: a seat assembly having a seat inwhich the user is to sit, and a power adjustable seat support assemblyconnected to the seat and configured to change a position or anorientation of the seat based on receiving a signal from the processor,wherein the processor signals the power adjustable seat support assemblyto automatically change the position or orientation of the seat based onthe detected head position of the user.
 10. The display system of claim1, further comprising: a power adjustable display support assemblyconnected to and supporting the display, and configured to change aposition or an orientation of the display based on receiving a signalfrom the processor, wherein the processor signals the power adjustabledisplay support assembly to automatically change the position ororientation of the display based on the detected head position of theuser.
 11. The display system of claim 1, wherein the processor is incommunication with a surgical robotic control system, and wherein theprocessor sends a signal to the surgical robotic control system to pausean operation of the surgical robotic control system when the gaze of theuser is not directed towards the display.
 12. The display system ofclaim 1, wherein a control panel having a plurality of icons isdisplayed on the display, and wherein an application related to adisplayed icon of the plurality of icons is invoked when the processordetermines that the detected gaze of the user is directed at thedisplayed icon.
 13. The display system of claim 1, wherein a position ororientation of the surgical robotics camera is changed based upon thedetected gaze of the user.
 14. The display system of claim 1, whereinthree-dimensional image data received from the surgical robotics camerais displayed on the display, and wherein when the processor determinesthat the detected gaze of the user is directed at an area or point onthe three dimensional image data that is spaced apart from a center ofthe display, the processor generates and sends a signal to the surgicalrobotics camera to adjust the position or orientation of the surgicalrobotics camera such that the area or point on the three-dimensionalimage data is moved so as to be centered along the display.
 15. Athree-dimensional display system for use in a surgical robotics system,comprising: an autostereoscopic three-dimensional display configured toreceive and display video from a surgical robotics camera; a pluralityof sensor assemblies that include a first sensor assembly and a secondsensor assembly; a processor configured to detect and track a headposition of a user relative to the display based on processing outputdata received from the first sensor assembly, and to detect and track agaze of the user based on processing output data received from thesecond sensor assembly, the processor generating control signals tomodify an operation of the display system based on the detected gaze ofthe user, and to automatically adjust a spatial relationship between theuser and the display based on the detected head position of the user toaffect the user's visualization of three-dimensional images on thedisplay.
 16. The display system of claim 15, wherein the displayincludes at least one of polarizing filters, a pattern retarder, ordynamic shutters, and wherein the user uses glasses or other wearablecomponents to view or visualize the three-dimensional images on thedisplay.
 17. The display system of claim 15, further comprising: a seatassembly having a seat in which the user is to sit, and a poweradjustable seat support assembly connected to the seat and configured tochange a position or an orientation of the seat based on receiving asignal from the processor, wherein the processor signals the poweradjustable seat support assembly to automatically change the position ororientation of the seat based on the detected head position of the user.18. The display system of claim 15, further comprising: a poweradjustable display support assembly connected to and supporting thedisplay, and configured to change a position or an orientation of thedisplay based on receiving a signal from the processor, wherein theprocessor signals the power adjustable display support assembly toautomatically change the position or orientation of the display based onthe detected head position of the user.
 19. The display system of claim15, wherein three-dimensional image data received from the surgicalrobotics camera is displayed on the display, and wherein when theprocessor determines that the detected gaze of the user is directed atan area or point on the three dimensional image data that is spacedapart from a center of the display, the processor generates and sends asignal to the surgical robotics camera to adjust the position ororientation of the surgical robotics camera such that the area or pointon the three-dimensional image data is moved so as to be centered alongthe display.
 20. A method performed by a digital programmed processorexecuting instructions stored in a computer readable memory, the methodcomprising: receiving video signals from a surgical robotics camera andrendering images on a three-dimensional display based upon the receivedvideo signals; detecting and tracking a head position of a user relativeto the display based on processing output data from a first sensor;detecting and tracking a gaze of the user based on processing outputdata from a second sensor; automatically signaling an actuator subsystemto adjust a spatial relationship of the display in relation to the userbased on the detected head position of the user to optimize the user'svisualization of three-dimensional images on the display; andautomatically signaling the actuator subsystem to adjust a position ororientation of the display or a seat assembly based upon the headposition of the user.